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1.0  INTRODUCTION 


This  final  report  describes  the  decision  analysis  work¬ 
shops  conducted  by  analysts  of  Decisions  and  Designs,  Inc. 
(DDI)  for  members  of  the  J-5  staff,  at  the  Command  and  Control 
Technical  Center  (CCTC).  The  intent  of  the  workshops  was  to 
apply  decision-analytic  support  to  current  J-5  problems  while 
providing  CCTC  with  the  knowledge  and  techniques  necessary  to 
support  future  J-5  decision  analysis  needs.  However,  some  of 
the  workshops  were  tutorials  on  decision  analysis  software, 
methodologies,  and  problems  that  DDI  has  encountered. 

This  report  is  a  chronological  listing  of  the  completed 
workshops  (1  July  1980  to  21  April  1981)  and  their  content 
with  emphasis  on  the  decision  analysis  techniques  applied  or 
discussed . 
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2 . 0  WORKSHOPS 


2 . 1  Worldwide  Digital  System  Architecture  Evaluation 

(Workshops  1  and  2) 

The  1  and  3  July  1980  workshops  were  attended  by  members 
of  the  Defense  Communications  Agency  (DCA)  and  representa¬ 
tives  from  Mitre  Corporation.  Their  task  was  to  evaluate 
nine  alternative  design  concepts  for  a  Worldwide  Digital 
System  Architecture  (WWDSA),  proposed  by  Mitre,  with  respect 
to  criteria  developed  by  DCA.  Prior  to  the  conference,  a  DCA 
analyst  had  structured  the  criteria  for  the  evaluation  in  a 
form  that  corresponded  to  DDI's  hierarchical  evaluation 
(HIVAL)  model. 

Using  this  structure,  the  analysts  at  the  conference 
focused  on  selecting  a  subset  of  criteria  for  the  evaluation 
process  and  on  assessing  the  values  of  the  alternatives. 
Because  the  meeting  was  restricted  to  two  days,  it  was  neces¬ 
sary  to  reduce  the  set  of  criteria  to  a  manageable  size.  For 
the  first  day's  analysis  the  original  tree  structure,  shown 
in  Figure  2-1,  was  pruned  from  52  to  17  bottom-level  criteria; 
the  revised  structure  is  shown  in  Figure  2-2.  The  group 
decided  to  address  only  "Effectiveness"  parameters  and  only 
in  a  broad  sense.  If,  during  the  evaluation  procedure,  the 
group  had  experienced  difficulty  with  broad  criteria,  more 
detail  could  have  been  added. 

For  the  second  day's  analysis,  a  new  set  of  criteria  was 
added  to  the  structure,  as  shown  in  Figure  2-3. 
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The  nine  alternative  design  concepts  were  scored  with 
respect  to  each  bottom-level  criterion,  and  the  criteria  were 
weighted.  The  resulting  overall  scores  for  the  analysis  are 
displayed  in  Figure  2-4.  System  Design  2  includes  variations 
2A,  26 ,  and  2C.  These  variations  received  overall  scores  of 
63,  59,  and  60,  respectively,  and  were  the  superior  design 
alternatives  or  options.  The  performance  of  these  options  on 
Effectiveness  and  Implementation  criteria  is  displayed  in  the 
matric  >  labeled  1  and  2. 

Figure  2-5  charts  the  options’  scores  on  Effectiveness 
versus  Implementation.  (Here  option  1  is  BL,  2  is  1A,  3  is 
IB,  etc.)  A  perfect  option  would  appear  in  the  upper  right- 
hand  corner.  Note  that  options  3A  and  3B  (8  and  9)  score 
very  high  on _ Effectiveness  but  would  be  the  most  difficult 
options  to  implement.  Options  2A,  2B,  and  2C  (5,  6,  and  7) 
dominate  all  others  when  both  criteria  are  considered. 

Figure  2-6  shows  how  the  options  fare  when  the  weight  of 
the  Implementation  criterion  varies.  System  Designs  2A,  2B, 
and  2C,  which  score  moderately  well  on  all  criteria,  appear 
consistently  strong  and  dominate  in  the  mid-range. 

As  a  result  of  this  analysis,  the  group  decided  to 
concentrate  on  System  Design  2  and  experiment  with  variations 
during  future  sessions. 


2.2  Decision  Structuring  Tutorial  (Workshop  3) 

DDI  analysts  met  with  three  CCTC  analysts  on  10  July  to 
discuss  the  aspects  of  structuring  decision  models.  The 
group  exercised  an  example,  "The  Rambo  Crisis,"  and  built  a 
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hierarchical  multi-attribute  evaluation  model  and  a  decision 
tree  model.  In  addition,  they  practiced  building  the  models 
on  the  computer. 

2 . 3  Analysis  of  a  Contingency  Problem  (Workshops  4  and  5) 

Two  workshops  were  held  on  7,  8,  and  11  August  to  address 
a  potential  contingency  problem  under  the  supervision  of 
Colonel  J.  D.  Beans.  The  analytical  approach  was  multi¬ 
attribute  utility  analysis:  first,  the  problem  and  its 
alternatives  were  identified.  Then  a  set  of  attributes  were 
developed  that  could  be  used  to  evaluate  the  alternatives. 
Because  the  output  of  this  analysis  was  deemed  classified, 
all  results  were  left  with  Colonel  Beans. 

2 . 4  Resource  Allocation  Software  (Workshop  6) 

A  DDI  analyst  met  with  CCTC  representatives  on  2  and  3 
September  at  their  offices  to  display  and  describe  two  DDI 
decision-analytic  software  packages  that  can  be  used  for 
resource  allocation  problems.  The  group  was  briefed  on  POM, 
the  software  that  supports  the  resource  allocation  method¬ 
ology  used  by  the  services  to  prepare  their  Program  Objecives 
Memorandum,  and  DESIGN,  another  DDI  resource  allocation 
methodology.  DESIGN  is  a  decision-analytic  technique  that  is 
useful  for  decisions  that  involve  choosing  a  set  of  optimal 
(cost-beneficial)  levels  of  various  items  that  compete  simul¬ 
taneously  for  resources. 

2.5  Middle  East/Africa  Division  Briefings  (Workshop  7) 

DDI  analysts  briefed  representatives  of  the  Middle 
East/Africa  Division  (MEAF),  J-5  on  19  September  1980.  The 
workshop  addressed  a  classified  problem  of  alternative  courses 
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of  action  selection  utilizing  hierarchical  multi-attribute 
utility  modeling.  During  the  course  of  the  workshop,  two 
decision  problems  of  the  resource  allocation  type  were  identi¬ 
fied  for  possible  analysis  in  future  workshops. 

2 . 6  Rapid  Deployment  Joint  Task  Force  (RDJTF)  Briefings/ 
Discussions  (Workshop  8) 


On  17  and  20  October,  DDI  analysts  attended  briefings 
and  held  discussions  with  RDJTF  representatives  to  identify 
decision  problems.  The  most  promising  problems  were  force 
structuring  decisions. 

2 . 7  Second  WWDSA  Evaluation  (Workshops  9  and  10) 

Representatives  from  the  Defense  Communications  Engi¬ 
neering  Center  and  Mitre  Corporation  returned  to  DDI  on  29 
October  and  5  November  for  a  second  decision  conference  to 
evaluate  a  new  set  of  alternate  architectural  system  designs. 
The  group,  led  by  DDI  analysts,  used  the  multi-attribute 
hierarchical  evaluation  model  (HIVAL)  to  analyze  eight  options. 
The  set  of  criteria  remained  the  same  as  before.  The  simplest 
or  baseline  option  was  a  design  derived  from  "Strawman  2"  of 
the  previous  analysis  and  was  entitled  2A.  The  other  options 
were  variations  of  the  basic  system,  i.e.,  with  one  or  more 
of  the  following  features  added:  Interoperability  (I), 

COMSEC  (C),  and  Satellites  (S).  For  instance.  Option  IS  is 
the  design  2A  plus  Interoperability  and  Satellites. 


Each  alternative  was  scored  on  each  criterion  relative 
to  all  other  alternatives,  and  the  criteria  were  weighted. 
Figure  2-7a  displays  the  resulting  scores  beneath  each  option 
on  the  row  called  TOTAL. 

Except  for  the  COMSEC-only  option,  the  alternatives 
score  similarly  when  EFFECTIVENESS  and  IMPLEMENTATION  are 
weighted  equally.  Matrices  1  and  2  show  the  scores  in  these 
two  overall  categories. 

Figure  2-7b  is  a  sensitivity  analysis  of  the  weight  on 
IMPLEMENTATION  (which  is  the  criteria  category  that  includes 
COSTS).  Mote  that  when  IMPLEMENTATION  has  a  weight  of  40  or 
less,  that  is,  when  one  IMPLEMENTATION  point  is  spent  and  the 
return  on  EFFECTIVENESS  is  judged  to  be  1.5  points  or  more, 
the  design,  ISC  is  superior.  In  terms  of  cost,  the  next  best 
item  is  IS;  S  follows,  then  2A.  The  graph  in  Figure  2-7c 
demonstrates  that  options  2A,  S,  IS,  and  ISC  (#1,  6,  5,  and 
7)  dominate  as  the  weight  of  IMPLEMENTATION  varies. 
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2.8  Sea-Based  Air  Study  Tutorial  (Workshop  11) 

This  tutorial,  on  7  November,  consisted  of  a  survey  of 
models  that  were  constructed  for  the  evaluation  of  sea-based 
air  alternatives: 

"Decisions  and  Designs,  Inc.  (DDI)  has  recently 
concluded  an  extensive  decision-analytic  modeling  effort 
for  the  Sea-Based  Air  Studies  Office  (SBASO).  Using 
various  decision  analysis  techniques,  DDI  has  helped 
SBASO  to  organize  and  understand  the  immense  volume  of 
data  that  has  been  generated  by  the  numerous  studies 
that  SBASO  has  performed.  DDI  has  sought  to  provide  a 
systematic  decision-analytic  framework  for  the  evalua¬ 
tion  of  sea-based  air  alternatives.  This  framework 
incorporates  both  the  results  of  SBASO  studies  and  the 
accumulated  knowledge  and  judgment  of  experts  in  sea- 
based  air. 

The  decision-analytic  models  described  below  offer 
more  flexibility  in  evaluation  than  do  operations  re¬ 
search  modeling  techniques.  Decision-analytic  models 
can  be  altered  rapidly  and  at  little  expense.  Changes 
in  assumptions  can  quickly  be  incorporated.  Although 
more  conventional  techniques  do  accommodate  change, 
changes  are  sometimes  quite  difficult  and  are  sure  to 
consume  considerable  additional  time  and  expense.  In 
fact,  simulation  studies  conducted  for  SBASO  have  con¬ 
sumed  several  years  and  millions  of  dollars,  and  changes 
in  assumptions  often  involved  months  of  waiting  for 
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results.  Decision  analysis  also  allows  for  the  inclusion  of 
I  factors  not  typically  incorporated  in  simulations,  such  as 

risks  and  cost  and  transition  uncertainties.  Factors 
considered  too  trivial  and  inconsequential  for  a  simulation, 
such  as  weather  interference,  required  resupply,  and  ability 

•  to  stage  and  base  from  alternative  platforms,  are  also 
easily  included. 

The  most  important  factor  underlying  the  choice  of 

•  sea-based  air  alternatives  is  the  operational  effectiveness 
of  the  options  being  considered.  The  DDI/SBASO  team  examined 
this  factor  from  two  perspectives.  The  first  perspective  is 
embodied  in  the  Mission  Area  Analysis  Model  (MAAM).  This 

t  model  compared  four  sea-based  air  alternatives: 

(1)  a  CTOL  force  based  on  CVN's  (CTOL-CVN); 

•  (2)  a  V/STOL  force  based  on  CVN's  ( V/STOL-CVN ) ; 

(3)  a  V/STOL  force  based  on  a  mixture  of  CVN's  and 
VSS’s  (V/STOL-CVN/VSS);  and 

» 

(4)  a  V/STOL  force  based  on  VSS ' s  ( V/STOL-VSS ) . 

(The  STOL,  STOVL ,  and  STOAL  options  were  not  evaluated, 

§  because  preliminary  decision-analytic  modeling  had 

demonstrated  that  these  approaches  were  not  cost  effec¬ 
tive,  tended  to  compound  the  Navy's  problems,  and  did 
not  contribute  to  the  Navy's  objective  of  operational 
{  flexibility.)  Each  alternative  was  evaluated  in  rela¬ 

tion  to  the  six  warfare  scenarios  identified  by  the 
Center  for  Naval  Analysis  (CNA).  For  example,  the 
scenarios  included  a  major  war  at  sea  in  the  Green- 
C  land-Iceland-Unite  i  Kingdom  Gap  and  power  projection  in 
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Korea.  The  MAAM  results  suggested  that  V/STOL-CVN/VSS 
is  best  in  operational  effectiveness,  CTOL-CVN  is  second, 
V/STOL-CVN  is  third,  and  V/STOL-VSS  is  worst. 

Although  the  MAAM  and  Operational  Effectiveness 
Submodel  (OES)  do  not  entirely  agree  on  the  ordering  of 
sea-based  air  alternatives,  they  are  close.  For  instance, 
both  suggest  that  V/STOL-CVN/VSS  is  best.  In  the  MAAM, 
this  represents  an  advantage  due  to  placing  small  platforms 
in  scenarios  that  do  not  require  a  large  CVN.  In  other 
words,  V/STOL-CVN/VSS  offers  flexibility  in  platform 
allocation.  In  the  OES,  the  preference  for  V/STOL-CVN/VSS 
represents  both  flexibility  and  the  operational  advantage 
of  V/STOL  over  CTOL.  The  MAAM  and  OES  also  agree  that 
V/STOL-VSS  is  the  worst  option.  In  both  cases,  this 
reflects  the  vulnerability  of  the  VSS .  As  for  the 
intermediate  options  (CTOL-CVN  and  V/STOL-CVN),  the  two 
models  disagree.  This  is  due  to  the  emphasis  that  the 
MAAM  places  on  aircraft  quantity,  which  favors  CTOL-CVN, 
and  the  emphasis  that  the  OES  places  on  aircraft  and 
platform  quality,  which  favors  V/STOL-CVN.  Although 
this  conflict  between  quantity  and  quality  is  not  fully 
resolved  in  these  models,  the  predominance  of  V/STOL-CVN/VSS 
in  both  models  suggests  that  V/STOL  is  a  viable  option. 

While  operational  effectiveness  is  the  most  impor¬ 
tant  criterion  for  judging  sea-based  air  alternatives, 
it  is  not  the  only  one.  To  capture  the  additional 
factors,  DDI  constructed  a  Comprehensive  Evaluation 
Model  (CEM)  that  considered  the  following  five  factors: 

(1)  Operational  Effectiveness; 

(2)  Transition  Difficulties; 
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(3) 

Cost  Factors; 

1 

(4) 

Risks;  and 

(5) 

Technological  Impacts. 

The  results  of  the  OES  were  used  to  score  the  first 
factor.  The  other  four  factors  were  then  evaluated  for 
each  of  the  four  sea-based  air  alternatives.  The  results 
of  the  CEM  suggested  that  V/STOL-CVN/VSS  is  best,  V/STOL- 
CVN  is  second,  CTOL-CVN  is  third,  and  V/STOL-VSS  is 
worst . 

Although  this  finding  parallels  the  OES  results,  it 
can  be  shown  that  the  results  are  quite  sensitive  to  the 
weight  placed  on  Risk.  At  present.  Operational  Effective¬ 
ness  and  Risks  are  weighted  41%  each.  If  the  weight  on 
Risk  were  increased  by  as  little  as  5%  in  relation  to 
Operational  Effectiveness,  CTOL-CVN  would  become  the 
preferred  option.  This  underscores  the  importance  of 
obtaining  accurate  estimates  of  the  risks  involved  in 
the  V/STOL  technology. 

The  three  models  described  thus  far  adhered  to  the 
ground  rules  laid  down  by  the  SBASO:  All  forces  were 
equal-cost  forces,  and  the  CTOL  and  V/STOL  designs  were 
assumed  equivalent  on  all  performance  factors  not  related 
to  take-off  and  landing.  A  number  of  additional  DDI 
models  allowed  these  stipulations  to  be  relaxed;  these 
models  are  called  Design  models.  Three  such  models  were 
constructed,  the  most  valuable  of  which  was  the  V/STOL 
Design  Model. 

The  V/STOL  Design  model  asks  whether  V/STOL  might 
be  made  less  expensive  without  loss  of  performance, 
whether  V/ST01  is  well  designed  for  accomplishing  a  mix 
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of  missions,  and  whether  it  might  be  advisable  to  design 
two  rather  than  one  V/ST01  aircraft.  The  model  embodies 
cost  and  benefit  assessments  for  aircraft  performance 
factors  (e.g.,  speed,  crew  size,  and  payload)  as  they 
relate  to  four  missions:  air-to-air  against  a  bomber, 
air-to-air  against  a  fighter,  air-to-surface  (all  weather), 
and  air-to-surface  (visual).  The  results  suggested  that 
V/STOl  could  be  redesigned  at  10%  less  expense  without 
loss  of  benefit,  that  the  current  proposed  design  favors 
the  air-to-air  against  a  fighter  mission  at  the  expense 
of  other  missions,  and  that  two  V/STOl  designs  tailored 
to  different  missions  could  potentially  decrease  the 
cost  by  10%  or  more  without  loss  of  benefit.  This  final 
result  is  only  speculative,  however,  because  the  V/STOL 
design  model  did  not  incorporate  the  increased  costs  of 
Research  and  Development,  maintenance,  or  training  for 
two  designs  rather  than  one. 

In  addition  to  the  substantive  models  described 
above,  DDI's  effort  on  behalf  of  SBASO  included  an 
Influence  Diagram  Model.  This  model  organized  the 
factors  relating  to  the  sea-based  air  decision  and 
documented  each  factor  according  to  the  studies  that 
provide  information  about  that  factor.  Thus,  the  Influ¬ 
ence  Diagram  model  offers  an  index  to  the  sea-based  air 
studies . 

Based  on  the  models  developed  by  DDI,  the  conclu¬ 
sions  and  recommendations  regarding  the  future  of  sea- 
based  air  are  straightforward.  V/STOL  is  an  attractive 
option,  especially  if  based  on  a  mixture  of  large  and 
small  platforms.  Besides  the  operational  flexibility 
offered  by  such  a  mixture  of  platforms,  V/STOL  is,  in 
its  own  right,  a  qualitatively  better  aircraft  than 
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CTOL .  This  qualitative  difference  is,  in  many  cases, 
sufficiently  great  to  overcome  the  quantitative  loss  of 
aircraft  due  to  V/STOL's  greater  cost.  Also,  there  is 
some  suggestion  that  the  quantitative  differences  could 
be  at  least  partially  ameliorated  by  a  more  efficient 
V/ST01  design. 

One  factor  that  argues  against  V/STOL,  in  favor  of 
CTOL,  and  therefore  demands  careful  attention,  is  the 
technological  risk  involved  in  the  V/STOL  technology. 

If  the  risks  are  substantially  less  than  the  current 
estimates,  then  V/STOL  is  clearly  favored.  If  they  are 
much  greater,  then  CTOL  is  favored.  The  present  esti¬ 
mate  is  at  the  break  point  for  proceeding  in  either 
direction.  This  suggests  that  a  more  valid  assessment 
of  risk  is  needed.  Since  this  can  only  be  obtained  by 
initiating  research  on  technologies  that  are  both  risky 
and  differentially  risky,  for  V/STOL  and  CTOL  it  is 
recommended  that  such  research  should  begin  immediately. 

Stated  so  crisply,  the  conclusions  belie  the  amount 
of  careful  thought  and  effort  that  is  captured  in  the 
models.  With  the  aid  of  experts  from  the  SBASO,  DDI  has 
attempted  to  capture  all  of  the  factors  that  might 
influence  one's  judgment  about  the  choice  of  sea-based 
air  alternatives.  Wherever  possible,  the  results  of 
other  studies  have  been  incorporated.  Issues  have  not 
been  neglected,  however,  simply  because  data  were  unavail¬ 
able.  In  these  cases,  the  judgments  of  experts  were 
taken  as  data,  and  they  may  be  open  to  debate.  It 
cannot  be  denied,  however,  that  the  issues  being  raised 
are  relevant.  The  models  are  a  strucutred  represen¬ 
tation  of  human  judgment,  a  fact  freely  admitted,  but  «• 
such  they  are  both  comprehensive  and  well  organized. 


» 


With  careful  scrutiny  of  this  decision-analytic  modeling 

•  effort,  factors  influencing  the  choice  of  sea-based  air 
alternatives  should  become  apparent  and  the  implications 
of  the  models  more  easily  understood." 

•  2.9  Program  Objectives  Memorandum  (POM)  Tutorial 

(VJorkshops  12  and  13) 

The  three  members  of  CCTC  attended  a  two-day  tutorial  at 

*  DDI  on  20-21  November  that  addressed  the  resource  allocation 
methodology  used  by  DDI  to  assist  the  USMC  with  their  Program 
Objectives  Memorandum  (POM).  The  tutorial  covered  the  contents 
of  a  recent  DDI  report  (then  in  draft): 

Kenneth  P.  Kuskey,  Kathleen  A.  Waslov,  and  Dennis 
M.  Buede .  Decision-Analytic  Support  of  the  United 
States  Marine  Corps*  POM  Development:  A  Guide  to 

*  the  Methodology.  Final  Report  PR  81-6-158.  McLean, 

VA:  Decisions  and  Designs,  Inc.,  1981. 

During  the  first  day,  a  DDI  analyst  described  the  general 

*  form  of  assistance  provided  to  USMC,  and  outlined  the  proce¬ 
dures  for  program  identification  and  prioritization.  This 
included  a  review  of  examples  from  the  1982-1986  POM  and  a 
discussion  of  the  USMC-DDI  interaction  in  the  process. 

•  The  second  day  of  the  session  was  spent  discussing  the 
underlying  mathematical  model  that  represents  the  problem, 
and  the  method  for  the  model's  application.  A  DDI  analyst 
described  the  axiomatic  conditions  necessary  for  the  quanti- 

f  fication  of  preferences  for  USMC  program  items,  and  the 

analytical  means  for  obtaining  a  cost-benefit  prioritization. 

The  group  worked  problems  that  utilized  different  tech- 

'  niques  for  the  assignment  of  numbers  to  items  that  represent 
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preferences  for  the  items,  and  discussed  procedural  problems 
that  are  encountered  by  analysts  applying  the  techniques. 


2.10  Cost  Forecast  Planning  Tutorial  (Workshop 


This  tutorial  on  25  November  began  by  discussing  a 
factory  cost  forecasting  model  and  ended  with  discussions  of 
military  applications. 


The  cost  model's  purpose  is  to  quickly  estimate  the 
resource  implications  of  marginally  changing  the  production 
levels  of  one  or  more  products  in  a  factory.  The  military 
analogy  would  be  to  estimate  the  resources  required  to  accom¬ 
plish  changes  to  force  structure.  Figure  2-8  illustrates 
that  the  estimate  made  by  the  cost  model  is  just  one  aspect 
of  the  information  a  decision  maker  would  need  to  analyze 
changes  in  production  level.  It  is,  however,  one  of  the  more 
time-consuming  estimates  to  make,  normally  taking  two  weeks 
at  the  factory  for  which  the  model  was  automated  by  DDI . 

With  the  model,  estimates  are  made  in  minutes  rather  than 
weeks . 


The  basic  elements  of  the  model,  products  and  cost 
categories,  are  illustrated  in  Figure  2-9.  As  shown  in 
Figure  2-10,  the  model  lets  the  user  describe  a  proposed 
schedule  of  changes  to  production  levels  for  the  products. 
From  this  schedule  of  changes,  the  model  computes  cost  esti¬ 
mates  for  factory  operation  which  can  be  displayed  in  terms 
of  unit  product  costs,  cost  category  budgets,  and  annual 
expense  by  product  or  cost  category.  Figures  2-11  and  2-12 
show  the  generic  types  of  cost  that  are  accounted  for  in  the 
model,  including  fixed  costs,  variable  costs,  mixed  fixed- 
variable  costs,  investment  costs,  and  one-time  start-up 
costs. 
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EXISTING  PRODUCTS 
PRODUCT  A 
PRODUCT  B 
PRODUCT  C 
PRODUCT  D 
PRODUCT  E 
PRODUCT  F 
PRODUCT  G 
PRODUCT  H 
PRODUCT  I 
PRODUCT  J 

POTENTIAL  PRODUCTS 
PRODUCT  K 
PRODUCT  L 
PRODUCT  M 
PRODUCT  N 
PRODUCT  F 


MANUFACTURING  EXPENSE 

PURCHASING 

SHIPPING 

COE  ENGINEERING 

LINE  MAINTENANCE  ENGINEERING 

STRATEGIC  ENGR  (INCL  TOOLING) 

OTHER  STRAT  (MAJOR  REARRANGE) 

FINANCE 

SYSTEMS 

RECORDS 

OPERATION  STAFF 
FACTORY  STAFF 
PERSONNEL  RELATIONS 
RELIABILITY  STAFF 
INSURANCE  AND  TAXES 
DEPRECIATION 

COMM  PEOPLE  COSTS  PRODUCTION 

COMM  PEOPLE  COSTS  EXP/SALARY 

TRANSPORTATION 

INACTIVE  AND  OBSOLETE  STOCK 

MISCELLANEOUS  OVERHEAD 

DIRECT  LABOR  **S(INCL  LAB  VAR) 

DIRECT  MAT •!.  **SUNC:L  VAR) 

PROD  FACTORY  EXPENSE  * 

CAPITAL  EQUIPMENT 
ADDITIONAL  BUILDING  SPACE 
CHANGES  TO  INVENTORY  LEVELS 
CHAGES  TO  PAYABLES 
CHANGES  TO  RECEIVABLES 
PROD  INTRO  *  (INCL  MOVE/REARR) 


Figure  2-9 

BASIC  ELEMENTS  OF  THE  COST  FORECAST  MODEL 


WHAT  IT  DOES 


Changes  to  Product  Volumes 


By 

By 

By 

Products 

Cost  Center 

Year 

(15) 

(30) 
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Figure  2-10 
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OF  THE  COST  FORECAST 

MODEL 

25 


MANUFACTURING  PLANT 
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(  Figure  2-11 

AGGREGATE  COSTS  INCLUDED  IN  COST  MODEL 
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HOW  YOU  BUILD  IT  &  MODIFY  IT 


STRUCTURE 


Specify  Products 

Specify  Production-Level  Break-Points 
Specify  Cost  Categories 
Revise  Any  of  Above  at  Any  Time 


NUMBERS 


i 


Variable 


—Base  cost 
-Unit-cost 


"adjusters" 


Mixed  Ongoing 

—Split  Variable  and  Fixed 

—Split  pie  across  products  for  Variable 

—Show  relative  changes  by  product  across 

PRODUCTION  LEVELS 

Mixed  One-Time 

—Investment  $  by  product  and  production  level 
Fixed  One-Time 

— $  BY  PRODUCT 

Escalation  Rates  by  Cost  Type 
Revise  Any  of  the  Above  at  Any  Time 


Figure  2-12 

DETAILED  COSTS  INCLUDED  IN  COST  MODEL 


Figure  2-13  gives  an  idea  of  how  the  model  is  used; 
typical  outputs  are  shown  in  Figures  2-14  and  2-15.  Figure 
2-16  summarizes  the  major  benefits  are  (1)  fast  response  to 
management;  and  (2)  a  nonaccounting  approach  to  estimation, 
based  on  managerial  judgments  about  relative  costs  rather 
than  accountant  estimates  of  absolute  costs.  The  military 
example  worked  by  the  CCTC  staff  concerned  a  hypothetical 
restructuring  of  the  relative  sizes  of  the  Air  Force,  Army, 
Navy,  Marine  Corps,  and  Rapid  Deployment  Force. 

2.11  Structuring  Design  Models  Tutorial  (Workshop  15) 

A  tutorial  was  given  on  26  November  1980  to  present 
information  on  some  of  the  important  concerns  in  structuring 
Design  models.  The  methodology  represented  by  the  Design 
software  aids  decision  makers  in  the  allocation  of  limited 
resources  among  competing  programs.  The  major  concerns  in 
structuring  a  Design  model  involve  providing  a  structure  in 
which  variables  under  consideration  are  independent. 

The  session  began  with  a  discussion  of  the  steps  in  the 
design  process:  model  structuring,  assessment  of  costs  and 
benefits,  calculation  of  results,  and  sensitivity  analyses. 
Then  several  sources  of  interaction  were  presented. 

o  Impossible  levels  -  Variables  are  defined  so  that 
certain  combinations  are  impossible. 

o  Cost  interaction  -  The  costs  of  the  levels  of  one 
variable  depend  on  the  levels  chosen  for  other 
variables.  For  example,  this  interaction  would  be 
present  in  a  model  containing  the  two  variables, 
number  of  systems  and  system  quality.  The  cost  of 
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HOW  YOU  USE  IT 


—Review  Model  at  Aggregate  Level 

—See  Cost  Ratios  at  Status  Quo 

—See  Effect  of  New  Production  Mix  on  Total 
Costs.,  Cost  Ratios,  Unit  Costs 

—Enter  Multi -Year  Production  Plan;  See 

*  Year-by-year  effects  on  cost  growth  by 
product  or  by  broad  category 

.  Year-by-year  budgeting  needs  by  cost  center 


Figure  2-13 

HOW  TO  USE  THE  COST  MODEL 
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OUTPUTS  OF  THE  COST  MODEL 
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Instant  response  to  management 


"Splits  the  pie"  rather  than 
"counts  the  beans" 


Depends  on  managerial  judgment, 

NOT  ACCOUNTING  RECORDS 


Figure  2-16 

BENEFITS  OF  THE  COST  MODEL 


a  greater  number  of  systems  depends  on  whether  they 
are  high  quality  or  low  quality. 

o  Benefit  interaction  -  The  benefits  of  the  levels  of 
one  variable  depend  on  the  levels  chosen  for  other 
variables . 

o  Systematic  interaction  -  A  design  variable  inter¬ 
acts  with  other  variables  such  that  the  structure 
of  the  decision  problem  depends  on  the  level  of 
that  variable. 

When  interactions  are  minor,  they  may  often  be  ignored 
without  significant  impact  on  the  results  of  the  model.  More 
serious  interactions  require  modification  in  the  structure. 
Four  corrective  measures  were  presented. 

o  Combining  variables  -  Interacting  variables  can 
often  be  combined  into  a  single  variable.  This 
method  works  well  when  the  interaction  is  between 
two  variables  only. 

o  Redefining  variables  -  For  more  serious  inter¬ 
actions,  it  is  often  possible  to  redefine  the  vari¬ 
ables  so  that  the  interaction  is  reduced. 

o  Combine  several  analyses  -  For  systematic  inter¬ 
actions,  it  may  be  helpful  to  perform  several 
analyses  conditional  on  the  level  of  the  offending 
variable.  These  analyses  may  later  be  combined  to 
obtain  an  overall  result. 

o  Model  interactions  -  When  there  is  sufficient  time, 
it  may  be  useful  to  develop  a  model  which  incorpo- 


rates  the  interactions  into  the  calculations  of  the 
model.  The  interaction  between  quantity  and  quality 
is  an  interaction  which  may  be  addressed  by  this 
method. 

Examples  describing  each  of  these  methods  were  presented. 

2.12  Probability  Assessment  Tutorial  (Workshop  16) 

The  tutorial  on  Probability  Assessment  covered  the 
following  outline  of  topics  on  3  December: 

I.  Philosophies  of  Probability 

A .  Bayes ian/ Subjective/ Per sona 1 i stic 

B.  Frequency  Approach — Von  Mises 

C.  Objectivistic — Fisher ,  Neyman-Pearson 

II.  Axioms  of  Probability 

III.  Algebra  of  Events 

IV.  Topics  in  Probability 

A.  Theorems 

B.  Clairvoyant's  Test 

C.  Conditional  Probability 

D.  Independence 

E.  Chain  Rule  and  Expansion 

F.  Bayes'  Rule 

G.  Probability  Trees 

V.  Assessing  Probabilities 
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A.  Heuristics  and  Biases 

1.  Representativeness 

2.  Availability 

3.  Adjustment  and  Anchoring 

B.  Assessment  Procedures 

1.  Measure  of  Uncertainty 

a.  Probability 

b.  Odds 

c.  Log  odds 

2.  Response 

a.  Value 

b.  Fractile 

C.  Proper  Scoring  Rules 

D.  General  Interview  Process 


2.13  Aiding  Tactical  Intelligence  Analysis  Tutorial 


( Workshops  17  and  18) 


The  work  on  tactical  intelligence  analysis  that  DDI  is 
performing  for  the  Army  Research  Institute  (ARI)  was  pre¬ 
sented  in  a  two-day  session  (4-5  December).  The  session 
focused  on  two  different  decision  aids  being  developed  to 
help  tactical  intelligence  analysts  determine  the  most  likely 
enemy  courses  of  action  in  a  particular  situation.  The 
different  decision  aids  incorporate  slightly  different  norma¬ 
tive  approaches  to  decision  making.  Session  participants 
analyzed  a  tactical  intelligence  problem  using  the  different 
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aids;  this  enabled  them  to  evaluate  the  applicability  of  such 
aids  (and  approaches)  to  their  working  environment. 

On  the  first  day,  session  participants  used  a  multi¬ 
attribute  utility  assessment  ( MAUA )  aid  called  ENCOA  (Enemy 
Courses  of  Action)  to  evaluate  four  potential  enemy  avenues 
of  approach  represented  in  a  tactical  intelligence  scenario. 
To  accomplish  this,  the  following  four  procedural  steps  were 
implemented . 

First,  each  participant  studied  the  scenario  and  then 
briefed  the  others  on  what  avenue  of  approach  he  considered 
best  for  the  enemy  and  why.  Although  the  participants  had 
similar  views,  they  did  arrive  at  different  conclusions  about 
the  most  likely  avenue  of  approach. 

Second,  each  participant  scored  each  approach  on  the 
twenty-four  factors  in  ENCOA  and  weighted  the  relative  impor¬ 
tance  of  the  factors  to  obtain  an  overall  score  for  each 
avenue  of  approach.  The  overall  scores  obtained  with  ENCOA 
generally  agreed  with  those  conclusions  obtained  in  Step  #1; 
again,  the  participants  arrived  at  different  conclusions 
about  the  most  likely  avenue  for  the  main  enemy  attack. 

These  results  indicate  that  ENCOA  was  accurately  reflecting 
the  position  of  each  participant. 

Third,  each  participant  predicted  the  scores  and  weights 
assigned  by  his  counterpart  prior  to  seeing  the  ENCOA  results 
These  predictions  were  much  poorer  than  anticipated  based  on 
the  length  of  the  participants'  discussion.  For  example,  one 
participant  predicted  that  his  counterpart  would  not  even 
consider  "Disposition"  when  scoring  the  four  enemy  approaches 
in  fact,  his  counterpart  considered  this  to  be  the  most 
important  factor.  The  participants  discussed  the  possible 
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reasons  for  this  breakdown  in  communication  at  considerable 
length.  Both  agreed  that  using  a  MAUA  data  sheet  with  clear 
definitions  of  the  factors  would  greatly  facilitate  communi¬ 
cation  in  many  of  the  problems  they  are  tasked  to  analyze 
within  their  working  environment. 

Fourth,  each  participant  estimated  the  relative  likeli¬ 
hood  of  each  avenue  of  approach  on  the  basis  of  each  of  the 
five  categories  of  factors  in  ENCOA.  All  estimates  were  in 
the  form  of  odds  estimates;  the  first  avenue  of  approach  was 
compared  to  each  of  the  other  three.  This  permitted  one  to 
evaluate  the  correspondence  between  the  overall  utility 
scores  resulting  from  ENCOA  and  the  overall  probability 
scores  resulting  from  asking  the  same  questions,  but  using  a 
different  rating  scale.  If  ENCOA  is  reflecting  the  true 
position  of  each  participant,  there  should  be  a  good  corres¬ 
pondence  between  the  overall  utility  and  probability  scores. 
This  was  indeed  found  to  be  the  case. 

On  the  second  day,  session  participants  used  a  Bayesian 
decision  aid  to  evaluate  the  likelihood  of  the  same  four 
courses  of  action,  but  not  on  the  basis  of  new  information. 
Procedurally ,  each  of  the  three  participants  sequentially 
received  ten  messages  describing  enemy  activity.  After 
receiving  each  message,  the  participants  individually  esti¬ 
mated  the  likelihood  ratios  and  posterior  odds  for  approach 
#1  versus  each  of  the  other  three  approaches.  This  permitted 
one  to  evaluate  (a)  the  similarity  in  the  participants' 
judgments  and  (b)  the  correspondence  between  the  participants' 
posterior  odds  after  each  message  with  those  prescribed  by 
Bayes'  Theorem,  assuming  conditionally  independent  data. 

In  general,  the  participants'  likelihood  ratios  and 
posterior  odds  were  quite  similar.  All  participants  thought 
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one  course  of  action  was  clearly  most  likely  on  the  basis  of 
the  ten  messages,  although  there  was  some  disagreement  on  the 
relative  likelihood  of  two  of  the  other  three  enemy  approaches. 
Furthermore,  the  posterior  odds  for  each  participant  were 
less  extreme  than  those  prescribed  by  Bayes'  Theorem.  This 
"conservatism"  finding  replicates  that  obtained  in  previous 
research.  One  finding  not  demonstrated  in  the  literature, 
however,  is  that  participants'  behavior  may  not  be  conserva¬ 
tive  even  if  their  judgments  are.  Two  of  the  three  partici¬ 
pants  indicated  that  they  would  notifiy  the  friendly  commander 
than  an  enemy  attack  was  imminent  after  the  sixth  message 
even  though  that  at  that  time,  neither  their  posterior  odds 
nor  Bayes'  Theorem  suggested  such  extreme  action. 

Both  the  Defense  Communications  Agency  (DCA)  and  the  DD1 
analysts  learned  a  great  deal  during  the  second  day.  This 
was  the  first  time  that  the  DCA  participants  had  been  given 
an  opportunity  to  address  a  problem  using  a  Bayesian  decision 
aid.  They  thought  it  unlikely  that  such  an  aid  would  be  used 
frequently  in  their  working  environment.  This  was  also  the 
first  time  that  the  DDI  analysts  had  pilot-tested  the  Bayesian 
aid  being  developed  for  ARI  to  help  tactical  intelligence 
analysts  revise  their  opinions  on  the  basis  of  incoming  data. 
Interaction  with  the  DCA  participants  indicated  that  they  (1) 
felt  it  would  have  been  easier  and  more  meaningful  working 
with  probabilities  rather  than  odds;  (2)  had  considerable 
difficulty  maintaining  the  operational  distinction  between 
likelihood  ratios  and  posterior  odds  and  between  conditional 
independent  and  dependent  data;  and  (3)  would  be  averse  to 
relying  on  the  posterior  odds  prescribed  by  Bayes  Theorem 
because  they  differed  so  much  from  their  own  estimates,  and 
it  was  not  at  all  clear  to  them  that  the  Bayesian  odds  were 
"better"  in  some  qualitative  sense.  Such  information  should 


prove  valuable  during  the  development  of  a  Bayesian  aid  for 
tactical  intelligence  analysis. 


2 . 14  Third  V7UDSA  Evaluation  (Workshops  19  and  20) 

The  analysis  at  the  third  tWDSA  decision  conference  at 
DDI  (8  and  9  December)  was  an  evaluation  of  eight  architec¬ 
tural  system  designs  which  represented  refinements  of  former 
design  variations.  Again,  the  baseline  system.  Option  A, 
could  be  made  more  effective  with  additional  features  that 
are  incorporated  in  Options  B  through  H.  However,  the  costs 
of  Implementation  for  the  enhanced  systems  are  higher,  thus 
they  score  lower  in  the  analysis  with  respect  to  Implementa¬ 
tion  criteria. 

The  alternatives  received  scores  for  each  criterion  in 
the  hierarchy  that  was  established  at  the  first  DCEC  confer¬ 
ence,  and  the  criteria  were  weighted.  Figure  2-17a  shows  how 
the  overall  scores  of  the  alternatives  vary  as  the  weight  of 
Implementation  changes.  IThen  its  weight  is  low  and  the 
Effectiveness  criteria  are  considered  to  be  the  best  discrimi¬ 
nators  for  the  options,  then  Options  G  and  H  dominate.  But 
as  the  costs  of  Implementation  enter  the  evaluation  as  signi¬ 
ficant  discriminating  factors,  Options  G  and  H  become  less 
attractive  because  they  are  expensive  and  risky  designs. 


Figure  2-17b  plots  Effectiveness  versus  Implementation 
and  demonstrates  that  all  eight  options  are  approximately 
equally  cost-effective,  i.e.,  spending  more  Implementation 
dollars  on  an  enhanced  design  will  return  a  proportionate 
amount  of  Effectiveness. 
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2.15  Utility  Assessment  Tutorial  (Workshop  21) 

On  11  December,  a  tutorial  attended  by  three  DCEC  analysts 
addressed  the  following  two  subjects: 

o  Assessing  Soft  Variables  -  The  Lawrence  Livermore 
Laboratory /Nuclear  Regulatory  Commission  (LLL/NRC) 
study  on  public  values  concerning  the  health  hazards 
of  nuclear  waste  was  reviewed.  Techniques  for 
assessing  "soft"  or  controversial  variables  such  as 
life,  death,  and  illness  were  examined  in  detail. 

o  Influence  Modeling  -  This  topic  addressed  how  to 
formulate  a  problem  and  structure  it  initially  in 
order  to  select  an  appropriate  analytical  approach. 
The  problem  addressed  was  developing  a  system  for 
prioritizing  military  targets.  After  specifying 
the  outcome  dimensions  of  principal  value,  we 
listed  some  of  the  conditioning  variables  and 
roughed,  out  the  form  that  a  working  model  might 
take.  The  participants'  response  was  quite  positive. 

2 . 16  Cascaded  Inference  and  Opinion  Revision  Tutorial 
(Workshop  22) 

On  30  December,  a  tutorial  was  held  on  hierarchical/ 
cascaded  inference.  This  tutorial  included: 

I.  Mathematical  Underpinnings 

A.  Extension  of  Bayes*  Rule 

B.  The  likelihood  principle 


i 


C 
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II.  Results  from  Empirical  Studies 


A.  "Lying  experimenter,"  bookbag  and  poker  chip 
tasks — excessiveness 

B.  Evaluation  of  evidence  from  court  trials — 
conservatism 


C.  Inappropriate  opinion  revision  of  subjects 

given  conflicting,  contradictory,  or  redundant 
data 


III.  Applications  of  Hierarchical  Inference  Methods 


A.  Indications  and  warning 

1.  Korean  warning  problem 

2.  Soviet  posture  model 

B.  Technical  intelligence 


IV.  Issues  for  Future  Application 

A.  Multi-valued  versus  binary  intermediate  vari¬ 
ables 

B.  User  responsiveness 

C.  Training 


2.17  Automated  Network  Targeting  Aid  (Workshops  23-48) 

During  January-April  1981,  a  series  of  twenty-six  meetings 
was  held  to  develop  design  concepts  for  a  new  military  targeting 
aid.  The  dates  of  the  26  meetings  were:  4,  10,  11,  19,  25, 
and  26  February;  3,  4,  6,  12,  13,  17,  18,  19,  20,  24,  27,  and 
31  March;  2,  7,  8,  10,  14,  16,  20,  and  21  April.  The  need 


for  a  new  aid  arose  from  urgent  problems  faced  by  the  target 
coordination  staff  of  the  Rapid  Deployment  Joint  Task  Force 
(RDF).  The  RDF's  target  coordinator  was  the  primary  "user" 
conferring  with  CCTC  and  DDI  in  the  meetings. 

The  decision  problem  faced  by  the  RDF  is  to  nominate 
targets  to  maximally  delay  a  larger  enemy's  advance  while  it 
projects  itself  into  operation.  The  targets  are  bridges, 
tunnels,  and  so  forth,  along  a  lines  of  communication  (LOC) 
network. 

The  decision-analytic  approach  taken  by  DDI  was  to  model 
the  network  mathematically  as  a  graph  and  then  devise  opera¬ 
tions  research  methods  for  maximizing  the  length  (duration) 
of  the  minimum  path  between  the  enemy  and  his  objectives. 

Both  the  graph  structure  and  data  were  built  to  represent  an 
expert  target  officer's  judgments  about  targeting  rather  than 
a  scientist's  description  of  the  problem.  The  model  is  then 
an  aid  that  will  help  the  target  officer  keep  track  of  his 
judgments  and  integrate  £hem  to  plan  targets.  It  is  not  a 
scientific  model,  but  a  judgment  model. 

During  the  meetings,  new  branch  and  bound  algorithms 
were  developed  for  maximizing  the  minimum  path  through  a 
network  by  selectively  damaging  targets.  Based  upon  these 
algorithms,  a  concept  for  an  automated  network  targeting  aid 
was  developed.  The  product  of  the  meetings  was  a  working 
report  (Advanced  Network-Targeting  Aid;  System  Design  Concepts . 
McLean,  Virginia:  Decisions  and  Designs,  Inc.,  April  1981) 
that  provides  design  concepts  for  a  state-of-the-art  targeting 
aid  incorporating  video  disc  technology.  These  concepts  form 
a  basis  for  additional  research  and  for  definition  of  functional 
requirements . 


